BOUNDARY BEHAVIORS AND SCALAR CURVATURE 
OF COMPACT MANIFOLDS 



YUGUANG SHI 1 AND LUEN-FAI TAM 2 



Abstract. In this paper, by modifying the arguments in [16], we 
get some rigidity theorems on compact manifolds with nonempty 
boundary The results in this paper are similar with those in [H] 
and [IB] . Like [Hj and [Hj, we still use quasi-spherical metrics 
introduced by [T] to get monotonicity of some quantities. 



In [Hj, the authors proved the following: Let {Q,g) be a compact 
manifold of dimension three with smooth boundary E which has pos- 
itive Gaussian curvature and has positive mean curvature. Suppose fl 
has nonnegative scalar curvature, then for each boundary component 
Sj of E satisfies, 



where H l Q is the mean curvature of Ej with respect to the outward 
normal when it is isometrically embedded in M 3 , dUi is the volume 
form on Ej induced from g. Moreover, if equality holds for some Ej 
then E has only one component and Q is a domain in M 3 . 

The result gives restriction on a convex surface E in R 3 which can 
bound a compact manifold with nonnegative scalar curvature such that 
the mean curvature of E is positive. It is interesting to see what one can 
say for convex surface E in H 3 k2 , the hyperbolic space with constant 
curvature — k 2 . 

The result mentioned above has other interpretation. It implies the 
quasi-local mass introduced by Brown- York [3l H] is positive under the 
condition that the boundary has positive Gaussian curvature. In [9], 
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[TO] . Liu and Yau introduced a quasi-local mass in spacetime. This 
quasi- local mass were also introduced by Epp [6] and Kijoswki [8]. 
More importantly, Liu and Yau proved its positivity, using [14J. A 
recent definition of quasi-local mass that relates with these works please 
see pi]. 

Motivated by [HI [91 [10] , in a recent work [16] Wang and Yau proves 
the following: Suppose (O, g) is a three dimensional manifold with 
smooth boundary E with positive mean curvature H, which is a topo- 
logical sphere. Suppose the scalar curvature 1Z of Q satisfies 1Z > —6k 2 
and the Gaussian curvature of £ is larger than — k 2 , then there is a 
future directed time-like vector value function W° on £ such that 



is time-like. Here H is the mean curvature of E when isometrically 
embedded in HP 2 , which is in turns isometrically embedded in M 3 ' 1 , 
the Minkowski space. In this result, the vector W° is not very explicit 
because it is obtained by solving a backward parabolic equation by 
prescribing data at infinity. 

In this work, by modifying the argument in [T6] , we get similar result 
by replacing W° by W So = (xi,x 2 ,x 3 ,at) for some a > 1 depending 
only on the intrinsic geometry of S. Here (x\, x 2 , £3, t) is the future 
directed unit normal vector of HP. 2 in 1R 3,1 . See Theorem 13 .11 for a more 
precise statement. We believe that the same result should be true with 
Ws = (xi,X2,Xs,t), but we cannot prove it for the time being. 

As a consequence, if o is a point inside of £ in HP 2 , then 



where r is the distance function from o in HP_ k2 . Moreover, equality 
holds if and only if (£l,g) is a domain in HP 2 - The results can be 
considered as generalization of the results in [14]. In fact, if we let 
k — > 0, we may obtain the inequality (11. ip . 

The paper is organized as follows. In §2, we list some facts that we 
need, most of them are from [16]. In §3, we prove our main results. 
We will also give some examples that a in Theorem 13.11 can be taken 
to be 1 and also study some properties of J^Hq — H) cosher oE. 



Most materials in this section are from Wang and Yau [16]. Let 
(Q, g) be a compact manifold with smooth boundary so that £ = dfl 
is a topologically sphere. Let H be the mean curvature with respect 





2. Preliminary 



Boundary behaviors and scalar curvature of compact manifolds 



3 



to the outward normal and K be the Gaussian curvature of S and let 
1Z be the scalar curvature of Q. In our convention, the mean curvature 
of the unit sphere in R 3 with respect to the outward normal is 2. By 
[T2l [5] , we have the following: 

Lemma 2.1. Suppose the Gaussian curvature K of S satisfies K > 
—k 2 . Then S can be isometrically embedded into the hyperbolic space 
HP K 2 with constant curvature —k 2 as a convex surface which bounds a 
convex domain D in H 3 k2 . Moreover, the embedding is unique up to 
an isometry o/H 3 ^. 

Since S is a topological sphere, its image S under the embedding 
divides HP_ 2 into two components, the exterior and the interior of S . 
Let N be the unit outward normal of S . It is said to be convex if the 
second fundamental form h(X, Y) = — (V^Y, N) is positive definite for 
X and Y tangent to So, where V K is the covariant derivative of H 3 k2 . 
The interior D of So being convex means that D is geodesically convex. 

The existence and uniqueness of the embedding were proved by 
Pogorelov [12]. The convexity of S and D were proved by do Carmo 
and Warner [5]. 

Further identify HP_ k2 with 

{(x 1 ,x 2 ,x 3 ,t) G R 3 ' 1 ] x\ + x\ + x\ - t 2 = — -, t > 0} 

where 1R 3,1 is the Minkowski space consisting of X = (xi, X2, x%, t) with 
the Lorentz metric dx\ + dx\ + dx\ — dt 2 . Position vectors in R 3 ' 1 can 
be parametrized as 

(2.1) 

X = (xi,x 2 ,x 3 ,t) 

= — (sinh nr cos 9, sinh nr sin 8 cos ip, sinh nr sin 6 sin ip, cosh nr). 

K 

The metric of H 3 2 is then 

dr 2 + k~ 2 sinh 2 nrda 2 = dr 2 + hT 2 sinh 2 Kr(d8 2 + sin 2 Qd^ 2 \ 

Note that r is the geodesic distance of a point from (0, 0, 0, 1/k) G H 3 2 - 
Let S p be the level surface outside So in H 3 k2 with distance p from 
S . Foliate H 3 k2 \ D by S p , p > 0. The hyperbolic metric can be 
written as ds 2 ^ = dp 2 + g p , where g p is the induced metric on S p . 

Suppose f F : S — > EI 3 _ k2 is the embedding with unit outward normal 
N. Then S„ subset of R 3 ' 1 is given by 

(2.2) X(p, p) = cosh np X(p, 0) + kT 1 sinh np N(p, 0) 
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Here for simplicity, (p, p) denotes a point S p which lies on the geodesic 
from the point p G S and X(p, 0) = X(F(p)). 

Suppose in addition that the mean curvature of if of £ with respect 
to (O, g) is positive and the scalar curvature 1Z of f2 is greater than 
or equal to —6k 2 . Wang and Yau |T6] are able to solve the following 
parabolic equation 



for all p > with positive and bounded solution u. Here A p is the 
Laplacian operator of E p , i? p is scalar curvature of S p , and if is the 
mean curvature of S p which is positive. We need the following result 
which is proved by Wang and Yau, see Theorem 6.1 and Corollary 6.1 



Theorem 2.2. [Wang- Yau] Let (Q, g) be a 3- dimensional compact Rie- 
mannian manifold with nonempty smooth boundary which is a topologi- 
cal sphere. Suppose the scalar curvature of 71 offl satisfies 71 > —6k 2 , 
the Gaussian curvature of its boundary £ satisfies K > —k 2 , and the 
mean curvature H of the boundary with respect to outward unit norm 
is positive. Let X be the position vector of H^ k2 in R 3 ' 1 then 



for any future directed null vector ( in M 3 ' 1 . 

Corollary 2.3. With the same assumptions and notations as in The- 
orem WTBf 



where r is as in ( TJQP - 

We also have the following rigidity result. 

Proposition 2.4. With the same assumptions and notations as in 
Theorem \2. 6 A Suppose 



for some future directed null vector ( in M 3 ' 1 , where the inner product 
is given by the Lorentz metric. Then Q is a domain in H 3 2 • 



(2.3) 




in [IS]. 





(2.4) 
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Proof. For simplicity, let us assume that k = 1. As in the proof of 
Theorem 12.21 in [16], let ds 2 = u 2 dp 2 + g p be the quasi-spherical metric 
where u is the solution of (12.31) . Let (M,g) be the manifold by gluing 
(Q,g) with H 3 X \ D with metric ds 2 . By [16], if (J23D is true, then 
the manifold (M, g) has a Killing spinor which is nontrivial, smooth 
away from E and is continuous. More precisely, satisfies: 



(2.5) V y + ^-^c(V) ■ (f) = 



where c(V)- is the Clifford multiplication. Hence M \ Q is Einstein. 
Since M has dimension three, the sectional curvature is —1 in (M\Q, g), 
see [2], for example. Let and fojj be the second fundamental form 
of So with respect to the metrics ds 2 ^ and ds 2 respectively. Then 

hij = u~ 1 h®j. By the Gauss equation and the fact that both ds^ and 

ds 2 have constant curvature —1, u = 1. 

On the other hand, <fi is not zero on So and so is a nontrivial 
Killing spinor in (Q, g) satisfying (12. 5ft and g has constant curvature 
— 1 as before. 

We claim that the second fundamental forms of S with respect to 
g and ds^ 3 are equal. If this is true, then by the proof of [HI Lemma 

4.1], we can conclude that g is smooth. From this it is easy to see that 
(M,g)=m. v 

Let us prove the claim. Since S is a topological sphere, for some 
a > a tubular neighborhood of S x (0, a) in g) is simply con- 
nected. Since it has constant curvature —1, (S x (Q,a),g) can be 
isometrically embedded in HP 1; see [H)J p. 43]. Denote the embedding 
by i = (mi, m 2 , m 3 ) where (ui, U2,Us) is are global coordinates in H^ 1 . 
Since i is an isometry, the normal curvatures of So x {r} for < r < a 
with respect to g and the hyperbolic metric are equal. Hence they are 
uniformly bounded on So x (0, a). Note that So is convex in HP 1; so 
S x {r} is also convex when embedded in HP 1 , for < r < a provided 
a is small. By [131 VI, §3], for any k > 0, | V^Uj| are uniformly bounded 
on So x (0, a), where V T is the covariant derivatives of So x {r} with 
induced metric by g. Hence by taking a subsequence of Tj — > 0, we 
obtain an isometric embedding of (E, g). In this embedding the sec- 
ond fundamental form with respect to g and ds 2 ^, are equal. Since 

the embedding of S is unique up to an isometry of H^, the claim is 
true. □ 
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3. Main results 

Let (Q,g) be as in Theorem 12.21 and let dQ = S. With the same 
notations as in the theorem, suppose o = (0, 0, 0,1/ k) is in D which is 
the interior of S in HP k2 C IR 3,1 . Let So be the image of S under the 
embedding described in §1. H is the mean curvature of So m HP K 2- 
We also identify S with S . Let 5 (i?i) and B Q (R 2 ) be geodesic balls 
in H 3 k2 such that B (R X ) C D C B (R 2 ). 

We want to prove the following: 

Theorem 3.1. Let (Q,g) be a compact manifold with smooth boundary 
S. Assume the following are true 

(i) The scalar curvature 1Z of (Q, g) satisfies 1Z > —6k 2 for some 
k > 0. 

(ii) S is a topological sphere with Gaussian curvature K > —k 2 and 
with positive mean curvature H . 

With the above notations, for any future directed null vector ( in 1R ' ; 



(3.1) m(fi;C) 



^(tf o -#)W So -CdS<0 



where Ws„ = (x±, x 2 , x%, at) with 



If sinh k.R 2 
a = coth kRi + -— — — — — t, 1 



sinh kRi \sinh 2 kR 1 

X = (xi, x 2 , X3, t) being the position vector in IR 3 ' 1 and the inner product 
is given by the Lorentz metric. Moreover, if equality holds in $3.1\) for 
some future directed null vector (, then (fl,g) is a domain in H 3 ^- 

Remark 3.2. If Ri,R 2 —>■ oo in such a way that R 2 — 1R\ — > — oo ; 
then a — > 1. 

In (12.11) . the position vector in M 3,1 is given by 
(3.2) X = (xi, x 2 , X3, t) = —((pi sinh nr, <\> 2 sinh nr, 03 sinh nr, cosh nr) 

K 

where (0i, 2 , 3 ) denote position vectors of points of S> 2 in IR 3 . Let {S p } 
be the foliation of HP 2 \ D described in §1. We need the following: 

Lemma 3.3. With the same assumptions and notations as in Theorem 
\3.1\ let yi, y 2 , j/ 3 GR with Y^=i Vi — 1; ^ e following are true. 
(i) For any p > 0, 

(3-3) £ > 

ap smh k/£ 2 
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(3.4) 

Hence 
(3.5) 

where 



J2i=i yi4>i, then for p> 

^k 2 sinh -2 nr ( 1 



.dr, 
dp' 



1 



dr 

dp ~ 



d<f) 



dp 



sinh kR\ 



sinh 2 kR 2 



- 1 



v sinh kR\ 

Proof, (i) Recall that o = (0,0,0, 1/ac) G D and r is the geodesic dis- 
tance in H 3 k2 from o. D is geodesically convex by Lemma 12.11 For 
any x, y G HP 2j denote the geodesic from x to y parametrized by arc 
length by xy. 

Let p G S and let 7(p) be the geodesic through p so that j(p) 
is orthogonal to £ at p with arc length parametrization. Moreover, 
7(0) = p and j(p) is outside E for p > 0. Let q be the point on 7 
such that d(o,q) = d(o, 7). Then g = 7(pi) with p x < because .D 
is geodesically convex. Since |^ = (Vr, Vp), |^ is nondecreasing on 
p > along 7 and |^ > at p. Let (3 = op and let r] be the geodesic 
from p on the totally geodesic HI 2 k2 containing 7 and (3 such that 77 is 
tangent to S . Let x, y be the intersection of rj with dB (R 2 ). Then 
> sin <p where <p is the angle between To and xp. Since 77 is outside 
B Q (Ri 



so 



sm<p 



> 



Hence we have 
(3.6) 



dr 



> 



sinh kR\ 
sinh kR 2 

sinh fti?! 



<9p sinh kR 2 
on H 3 2 \ D. 

(ii) Since the inner product in R 3 of (3/1,2/2,3/3) and (01,02,03) is 0, 
we may assume that = cos# in (12. lj) . The hyperbolic metric outside 
D is given by 



dp 2 + g p = dr 2 H — - sinh 2 nr(d9 2 + sin 2 9dip^ 



Compute ds 2 



e 3 



-JM in the above two forms of ds 2 ,,? , we have 



>■ dp ' <9p 



H3 ' 



,<9r\ 
dp' 



'H — - sinh «r 



dp' 



sm 



^p' 



> 



,<9r\ 
dp' 



2 +- 



smh fix( — , 
dp 
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Since <fi = cos 9, (ii) follows. 

The last assertion follows from (i), (ii), the fact that \<p\ < 1 and the 
fact that r > Ri for p > 0. 

□ 

Lemma 3.4. With the same assumptions and notations as in Theorem 

Hq-^-X + A„X - 2k 2 X = 
op 

in H1 k2 \ D. 

Proof. In the representation of HP 2 in (13.21) . the Laplacian of HP 2 is 
given by 

d 2 d 
A = 7— + 2k coth kt— — V k 2 sinh -2 nr A§2 . 
or 2 or 

So AX = 3k 2 X. In the foliation (ET2jl . the metric of HP k2 is given 

by dp 2 + g p where g p is the induced metric on level surface S p . The 

Laplacian on HP 2 is given by 



Using (12.21) . we have 



A 92 rr 9 a 



„ „ 9 



3k 2 X = + i/o^X + A p X 

(3.7) 

= K 2 X + #07T X + a p x - 
c/p 

From this the result follows. □ 

Now we are ready to prove Theorem 13. 11 
Proof of Theorem \3.1[ By [16] and Lemma 13.41 
(3.8) 

/ (H -H)XdX p 



d 
dp 



[ -\u-\u - 1) 2 (R P + 2k 2 )X + (u - 1) \ —^-X + A p X - 2k 2 x] dE, 
is p 2 \ u op J 



u-Hu-l) 2 



1 -(RP + 2k 2 )X + H^X 



dZp. 



Let \ a (p, p) be the principal curvature of the level surface. Then A a = 
Ktanhn(fjL a + p), k or /?coth/«(// a + p) with p a > 0, see [16]. Hence 
H Q = Ai + A 2 and i? p + 2k 2 = 2AiA 2 . Let W = a cosher and W = 
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0sinh kt where = Y^xVi^h Y^=iVi = 1 an d a * s the constant in 

the theorem. Then 

(3.9) 

dr 

A1A2 cosh nr + k(X\ + A2) sinh nr ■ — 

Op 



^(R p +2k 2 )W +H (W ) p = a 



(3.10) 

\(R P + 2k 2 )W + H W P = AiA 2 0sinhfi:r+ 

k(\i + X 2 ) ( 4> cosh kt • — — I — sinh nr ■ — — ) . 

\ Op K Op J 

Combining this with by Lemma [3. 3 [ (13.21) . (13. 8p and the fact that r > 
Ri in H 3 ^ \ D, we have 

(3.11) / (H o -H)(W-W o )dE p >0. 

By Theorem 12.21 and Corollary 12. 3\ we conclude that (13. ip is true. 

Suppose equality holds in (13.11) for some future directed null vector 
C, then using Corollary 12.31 we have 

(3.12) lim / (H -H)X- CdL p = 0. 

Jt, p 

By Proposition 12.41 (Q,g) is a domain in HP k2 . □ 

Remark 3.5. A3. 11\) means that for any future directed null vector (, 
d f 

(3.13) — (H -H)Wx o -(dZ p >0. 

From the proof of 113.11}) , it is easy to see that 113.13]) is still true if ( 
is future directed time-like. 

Corollary 3.6. With the same assumptions and notations as in The- 
orem [3~T[ Then for any p > 0, the following vector is either zero or is 
future directed non space-like: 

m(p) = / (H - H)W So dZ p . 
In particular, if (£l,g) is not a domain in HP_ k2 , then 

m(fi) = ^(H -H)W So dZ 
is future directed time-like. 
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Proof. By the proof of Theorem 13.11 and the characterization of future 
directed non space-like vector [16], we conclude the first part of the 
corollary is true. If (Q, g) is not a domain in H^ k2 , by the rigidity part 
of the theorem, this vector cannot be zero and cannot be null. Hence 
it is future directed time-like. □ 

Corollary 3.7. With the same assumptions and notations as in The- 
orem \37f\ let 



m(p) = / {H - #)W Eo dE p . 



then (|m(p)| 2 ) > ; where |m(p)| is the Lorentz norm ofm(p). 



Proof. For any fixed p , let ( be the vector m(p ). By Corollary 13.61 as 
mentioned above, ( is a future directed non space-like, note that 



fawn 



P=P0 



= 2(- / (H - iJ)W So • CrfS 

a P JT, t 



•c 

P=P0 



pJ\p=po 



By Remark 13.51 we have 



^(|m(p)| 2 )>0. 



□ 



Theorem 3.8. Let (£l,g) be as in Theorem \3.1\ and let So be the image 
of isometric embedding of dfl — E in 2 which encloses D. Then 
for any o G E : 



(H - H)(y) cosh rzr(o, y) dE (y) > 

'So 

Equality holds for some o if and only if (Q, g) is a domain of H 3 _ K 2 ■ In 
particular, if E is a standard sphere, then 

(H o -H)(y)dZo(y)>0 

and equality holds if and only if (Q, g) is a domain of H^ k2 . 

Proof. We may embed HP^ in IR 3 - 1 such that o = (0,0,0,1/k). The 
result then follows from Theorem 13.11 □ 

Theorem 13.11 implies a previous result of the authors [1 
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Theorem 3.9. Let (Q, g) be a compact three manifold with nonnegative 
scalar curvature and with smooth boundary E. Suppose E has positive 
Gaussian curvature and positive mean curvature H , then 



where H is the mean curvature of E when it is isometrically embedded 



This result follows from Theorem 13.81 and the following lemma. 

Lemma 3.10. With the same assumptions and notations as in The- 
orem \3.9[ suppose H K is the mean curvature of E when it is isomet- 
rically embedded in HP k2; k > 0. Then there exists Ki — > such that 
Hindoo E Ki = H . 

Proof. Let HP 2 be represented by the metric: 



defined on x\ + x\ + x\ = \x\ 2 < k~ 2 . We may assume that (E, g) 
is embedded to HP k2 with embedding i K such that p is mapped to 
the origin, where p is some fixed point. Then it is easy to see that 
E C B K (0, 2d) where d the intrinsic diameter of E and B K is the geodesic 
ball with respect to the metric (I3.14p . 

Let i K = «2,k, M3,k) in terms of the global coordinates xi, X2, x%. 
Since the Gauss curvature of E is positive, by [T3J VI§2,§3], we conclude 
that for any k > 0, for < k < 1 and 1 < j < 3, |V uniformly 
bounded. There exists Ki — > such that i K% together its derivatives 
converge to an embedding of E in the Euclidean space. Using the fact 
that the embedding of E in M. 3 is unique, it is easy to see that the 
lemma is true. □ 

Finally, we would like to give some examples to illustrate that in 
certain situations, a in Theorem 13.11 can be chosen as 1. The proofs 
of these examples are direct application of Theorem 13.11 Corollary 13.81 
and the representation of X given by (13.21) . 

Example 3.11. With the same assumptions and notations as in The- 
orem l37l\ ifHa standard sphere and H is constant, then m(Q,() — 
with W 2o = (xi, x 2 , x 3 ,t), for all future directed null vector £. Here we 
have assumed that (0,0,0,1/k) is inside E , which is the image of E 
under the embedding described in§l. 




(3.14) 



4 



-^{dx\ + dx\ + dx\) 



(1 - k- 
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Example 3.12. With the same assumptions and notations as in The- 
orem \3.1l if S a standard sphere its mean curvature H is orthogo- 
nal to the first eigenf unctions of SS 2 , then m(Q, () < with Ws = 
(xi,x 2 ,x 3 ,t), for all future directed null vector (. Here we assume that 
(0, 0, 0, 1/k) is the center of the geodesic ball in HP_ k2 enclosed by E , 
which is the image of S under the embedding described in §-/. 

Remark 3.13. It is easy to see that in Example \3.11\ and Example 
if (£l,g) is not a domain of H 3 , then on all of their small per- 
turbations, m(0, () < with W So = (xi,x 2 ,x 3 ,t). For perturbations 
of Example \ 3.11\ we only assume that (0,0,0, is inside S . For 
perturbations of Example \3.1B. the inequality is true if (0, 0, 0, is 
at some particular position inside So . 

Let f{p) = J s (H — H){y) cosh K,r(p,y) dS (?/), then it is a smooth 
function on D C H 3 ^, and it would be interesting to see some prop- 
erties of this function. We first have: 

Proposition 3.14. Suppose f has a critical point o in the interior of 
D, and let o = (0, 0, 0, 1/k), then for 1 < i < 3, 



Proof. It is easy to see that V G r(o, y) is the unit tangential vector of 
geodesic yo, hence, 

V r (o, y) = (cos 9, sin 9 cos ip, sin 9 sin i[>, 0) 

= (01,02,03,0) 

Combine this fact and a direct computation, we see that the conclusion 
is true. □ 

Remark 3.15. Suppose (Q,g) satisfying the assumptions in Theorem 
\3.1\ and o = (0, 0, 0,1/ k) be a critical point of f , then Theorem \3.1\ is 
true with a = 1. 

Again by a direct computation we have 

Proposition 3.16. Let f be defined as above, then 



Remark 3.17. Suppose (Q,g) satisfying the assumptions in Theorem 
\3.1\ then by maximal principle, we know that f cannot attain a local 
maximum inside of D; if there is o G D with f = 0, then f is identical 
to on the whole D which implies (Q,g) is a domain o/H 3 ^- 




(Ho — H) sinh nr ■ <pi = 0. 



Af = 3k 2 f 
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